Evaluation of the OQuaRE framework for ontology quality

International audienceThe increasing importance of ontologies has resulted in the development of a large number of ontologies in both coordinated and non-coordinated efforts. The number and complexity of such ontologies make hard to ontology and tool developers to select which ontologies to use and reuse. So far, there are no mechanism for making such decisions in an informed manner. Consequently, methods for evaluating ontology quality are required. OQuaRE is a method for ontology quality evaluation which adapts the SQuaRE standard for software product quality to ontologies. OQuaRE has been applied to identify the strengths and weaknesses of different ontologies but, so far, this framework has not been evaluated itself. Therefore, in this paper we present the evaluation of OQuaRE, performed by an international panel of experts in ontology engineering. The results include the positive and negative aspects of the current version of OQuaRE, the completeness and utility of the quality metrics included in OQuaRE and the comparison between the results of the manual evaluations done by the experts and the ones obtained by a software implementation of OQuaRE

Analysis of shared heritability in common disorders of the brain

ience, this issue p. eaap8757 Structured Abstract INTRODUCTION Brain disorders may exhibit shared symptoms and substantial epidemiological comorbidity, inciting debate about their etiologic overlap. However, detailed study of phenotypes with different ages of onset, severity, and presentation poses a considerable challenge. Recently developed heritability methods allow us to accurately measure correlation of genome-wide common variant risk between two phenotypes from pools of different individuals and assess how connected they, or at least their genetic risks, are on the genomic level. We used genome-wide association data for 265,218 patients and 784,643 control participants, as well as 17 phenotypes from a total of 1,191,588 individuals, to quantify the degree of overlap for genetic risk factors of 25 common brain disorders. RATIONALE Over the past century, the classification of brain disorders has evolved to reflect the medical and scientific communities' assessments of the presumed root causes of clinical phenomena such as behavioral change, loss of motor function, or alterations of consciousness. Directly observable phenomena (such as the presence of emboli, protein tangles, or unusual electrical activity patterns) generally define and separate neurological disorders from psychiatric disorders. Understanding the genetic underpinnings and categorical distinctions for brain disorders and related phenotypes may inform the search for their biological mechanisms. RESULTS Common variant risk for psychiatric disorders was shown to correlate significantly, especially among attention deficit hyperactivity disorder (ADHD), bipolar disorder, major depressive disorder (MDD), and schizophrenia. By contrast, neurological disorders appear more distinct from one another and from the psychiatric disorders, except for migraine, which was significantly correlated to ADHD, MDD, and Tourette syndrome. We demonstrate that, in the general population, the personality trait neuroticism is significantly correlated with almost every psychiatric disorder and migraine. We also identify significant genetic sharing between disorders and early life cognitive measures (e.g., years of education and college attainment) in the general population, demonstrating positive correlation with several psychiatric disorders (e.g., anorexia nervosa and bipolar disorder) and negative correlation with several neurological phenotypes (e.g., Alzheimer's disease and ischemic stroke), even though the latter are considered to result from specific processes that occur later in life. Extensive simulations were also performed to inform how statistical power, diagnostic misclassification, and phenotypic heterogeneity influence genetic correlations. CONCLUSION The high degree of genetic correlation among many of the psychiatric disorders adds further evidence that their current clinical boundaries do not reflect distinct underlying pathogenic processes, at least on the genetic level. This suggests a deeply interconnected nature for psychiatric disorders, in contrast to neurological disorders, and underscores the need to refine psychiatric diagnostics. Genetically informed analyses may provide important "scaffolding" to support such restructuring of psychiatric nosology, which likely requires incorporating many levels of information. By contrast, we find limited evidence for widespread common genetic risk sharing among neurological disorders or across neurological and psychiatric disorders. We show that both psychiatric and neurological disorders have robust correlations with cognitive and personality measures. Further study is needed to evaluate whether overlapping genetic contributions to psychiatric pathology may influence treatment choices. Ultimately, such developments may pave the way toward reduced heterogeneity and improved diagnosis and treatment of psychiatric disorders

Framework basado en el estándar de calidad del software ISO/IEC 25000:2005 (SQuaRE) para la evaluación de la calidad de las ontologías

Objetivos El objetivo general de esta tesis es crear un método de evaluación de la calidad de ontologías, adaptando el estándar de calidad del software ISO/IEC 25000:2005 Software Quality Requirements and Evaluation SQuaRE, que permite la trazabilidad entre requisitos y métricas de calidad de una ontología, con el fin de valorar de forma objetiva y reproducible sus características, y brindar así un mecanismo de asistencia a los usuarios y desarrolladores para la toma de decisiones informadas. Metodología 1. Se analiza el estado del arte relacionado con ontologías, lenguajes de representación de ontologías, evaluación de ontologías, y estándares de calidad del software y métricas de software. 2. Se diseña el framework OQuaRE para evaluación de la calidad de ontologías con base en SQuaRE. Para ello se definen (1) el modelo de calidad; (2) las métricas de calidad; (3) las asociaciones entre métricas y subcaracterísticas de calidad; y (4) la escala para las métricas, en el rango [1-5]. 3. Aplicación del framework a ontologías de diferentes dominios en el ámbito de ontologías biomédicas. 4. Validación del framework a través de la comparación de sus resultados con resultados de otros métodos de evaluación de ontologías como Gold Standard, Competency Questions, evaluación humana y evaluación de versiones. 5. Evaluación del framework por un grupo de expertos externos. Resultados 1. Un framework para evaluar la calidad de ontologías OQuaRE, el cual se compone de: (1) el modelo de calidad de ontologías que consta de un grupo de características y subcaracterísticas de calidad; (2) un conjunto de métricas de calidad; (3) las asociaciones subcaracterística-métrica; (4) las tablas de la escala de valores [1-5] para las métricas de calidad. 2. La aplicación del framework a un conjunto de ontologías de diferentes dominios con el fin de (1) identificar la capacidad de OQuaRE para detectar diferencias entre una ontología original y su versión normalizada; (2) identificar las fortalezas y debilidades de un conjunto de ontologías de unidades de medida, en términos de características y subcaracterísticas de calidad de OQuaRE; (3) identificar la utilidad de OQuaRE para evaluar métodos y guías de buenas prácticas de construcción de ontologías; (4) evaluar el impacto de la evolución de una ontología en su calidad identificando el efecto de los cambios entre versiones; (5) las características, subcaracterísticas y métricas aplicables a las ontologías financieras. Conclusiones OQuaRE es un framework de evaluación de ontologías que integra las medidas de varios criterios de la ontología y las agrupa en un modelo de calidad basado en el estándar de calidad del software SQuaRE ISO 25000:2005. OQUaRE evalúa la calidad de las ontologías de manera objetiva y reproducible, e incluye trazabilidad entre características, subcaracterísticas y métricas, partiendo desde un alto nivel, hasta llegar a un nivel más bajo de granularidad. OQuaRE contribuye a la toma de decisiones informadas porque presenta las fortalezas y debilidades de una ontología en términos de características y subcaracterísticas de calidad. OQuaRE sirve como base para evaluar la estructura de una ontología, desde su aspecto funcional hasta su aspecto más técnico, y puede ser de aplicación para ontologías de diferentes dominios. OQuaRE es un framework adaptable que permite incorporar nuevas métricas y asociaciones subcaracterística-métrica y crear o redefinir las escalas para cada repositorio particular. Las asociaciones característica-subcaracterística-métrica permiten un mejor entendimiento de las conexiones entre las características de una ontología y los criterios de calidad, además de una visión más clara de las métricas y su utilidad en la evaluación de las ontologías. Objectives The main goal of this thesis is to create a method for the evaluation of the quality of ontologies by adapting the software product quality standard ISO / IEC 25000: 2005 known as SQuaRE (Software Quality Requirements and Evaluation), which enables the traceability between requirements and quality metrics of an ontology, in order to evaluate ontologies in an objective and reproducible manner, and providing a mechanism for assisting users and developers making informed decisions. Methodology 1. Analysis of the state of the art in ontology representation, ontology languages, ontology evaluation, standards of software quality and software metrics. 2. Definition of the OQuaRE framework for ontology quality evaluation, based on SQuaRE, consisting (1) a quality model based on characteristics and sub-characteristics; (2) quality metrics; (3) associations between metrics and sub-characteristics; and (4) scale of the metrics’ scores to the range [1-5]. 3. Application of the OQuaRE framework to different domains ontologies (biomedical ontologies, cell types, units of measure). 4. Validation of the framework by comparing its results with those obtained by other methods of ontology evaluation such as Gold Standard, Competency Questions, human manual evaluation and evaluation of ontologies versions. 5. Evaluation of the method by a group of external experts. Results 1. A framework for evaluation the quality of ontologies OQuaRE, which consists in (1) the ontology quality model, with a set of quality characteristics and sub-characteristics; (2) a set of quality metrics; (3) list of associations subcharacteristic-metric; (4) the scales for transforming the scores of the metrics to the [1-5]. 2. The application of the framework to a set of ontologies from different domains has permitted to (1) identify the OQuaRE capacity to detect differences between different ontologies; (2) identify the strengths and weaknesses of sets of ontologies of the same domain, in terms of quality characteristics and sub-characteristics; (3) identify the usefulness of OQuaRE to evaluate methods and best practice guidelines for building ontologies; (4) evaluate the impact of the evolution of an ontology in its capacity to identify the effect of changes between versions; and (5) identify a set of characteristics, sub-characteristics and metrics useful for financial ontologies. Conclusions OQuaRE is a framework for evaluating ontologies which integrates several measures and criteria of ontology quality and groups them into a quality model based on the software quality standard ISO/IEC 25000: 2005 SQuaRE. OQUaRE evaluates the quality of ontologies in an objective and reproducible way, and includes traceability between characteristics, sub-characteristics and quality metrics, which permits to analyse the properties of an ontology at different granularity levels. OQuaRE contributes to informed decision making because it identifies the strengths and weaknesses of an ontology in terms of quality characteristics and subcharacteristics. OQuaRE serves as a basis for assessing the structure of an ontology, ranging from functional aspects to technical ones, and it can be applied for ontologies from different domains. OQuaRE is a customisable framework that allows to incorporate new metrics and associations subcharacteristic-metric, to create or redefine scales for each community. The associations Feature-subcharacteristic-metric enable a better understanding of the links between the characteristics of an ontology and quality criteria, along with a clearer view of the metrics and their useful in evaluating ontologies

OQuaRE: A square-based approach for evaluating the quality of ontologies

The development of the Semantic Web has provoked an increasing interest in the development of ontologies. There are, however, few mechanisms for guiding users in making informed decisions on which ontology to use under given circumstances. In this paper, we propose a framework for evaluating the quality of ontologies based on the SQuaRE standard for software quality evaluation. This method requires the definition of both a quality model and quality metrics for evaluating the quality of the ontology. The quality model is divided into a series of quality dimensions or charac-ter istics, such as structure or functional adequacy, which are organized into subcharacteristics, such as cohesion or tangledness. Thus, each subcharacteristic is evaluated by applying a series of quality metrics, which are automatically measured. Finally, each characteristic is evaluated by combining values of its subcharacteristics. This work also includes the application of this frame-work for the evaluation of ontologies in two application domains

Supporting the analysis of ontology evolution processes through the combination of static and dynamic scaling functions in OQuaRE

BACKGROUND: The biomedical community has now developed a significant number of ontologies. The curation of biomedical ontologies is a complex task and biomedical ontologies evolve rapidly, so new versions are regularly and frequently published in ontology repositories. This has the implication of there being a high number of ontology versions over a short time span. Given this level of activity, ontology designers need to be supported in the effective management of the evolution of biomedical ontologies as the different changes may affect the engineering and quality of the ontology. This is why there is a need for methods that contribute to the analysis of the effects of changes and evolution of ontologies. RESULTS: In this paper we approach this issue from the ontology quality perspective. In previous work we have developed an ontology evaluation framework based on quantitative metrics, called OQuaRE. Here, OQuaRE is used as a core component in a method that enables the analysis of the different versions of biomedical ontologies using the quality dimensions included in OQuaRE. Moreover, we describe and use two scales for evaluating the changes between the versions of a given ontology. The first one is the static scale used in OQuaRE and the second one is a new, dynamic scale, based on the observed values of the quality metrics of a corpus defined by all the versions of a given ontology (life-cycle). In this work we explain how OQuaRE can be adapted for understanding the evolution of ontologies. Its use has been illustrated with the ontology of bioinformatics operations, types of data, formats, and topics (EDAM). CONCLUSIONS: The two scales included in OQuaRE provide complementary information about the evolution of the ontologies. The application of the static scale, which is the original OQuaRE scale, to the versions of the EDAM ontology reveals a design based on good ontological engineering principles. The application of the dynamic scale has enabled a more detailed analysis of the evolution of the ontology, measured through differences between versions. The statistics of change based on the OQuaRE quality scores make possible to identify key versions where some changes in the engineering of the ontology triggered a change from the OQuaRE quality perspective. In the case of the EDAM, this study let us to identify that the fifth version of the ontology has the largest impact in the quality metrics of the ontology, when comparative analyses between the pairs of consecutive versions are performed

Significance levels of testing difference of means.

<p>Differences between means of untrained and trained groups () and mean distances to the gold standard of trained and untrained groups []: The character * is used to indicate the significance level of the differences: * significant, ** very significant and *** highly significant.</p

Significant effect of training in some topics.

<p>22 subcharacteristics presented significant effect due to the GoodOD based training for some topics (PRO, IMM, CLO, CME, INF, SPA).</p

No significant effect of training in any topic.

<p>Seven OQuaRE subcharacteristics presented no significant effect of the training for any topic (PRO, IMM, CLO, CME, INF, SPA), and their mean values were similar for students and the gold standard.</p

P-values associated with the main effects in the two-way ANOVA with no effect of the interaction training by topic.

<p>P-values associated with the main effects in the two-way ANOVA with no effect of the interaction training by topic.</p